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^ How  does  information  selection  differ  across  sites  in  the  text  and  graphics?^ 
The  results  of  this  experiment  provide  preliminary  answers  to  these  questions 
and  serve  as  the  basis  for  our  continuing  research  in  this  area. 
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ABSTRACT 


This  research  Is  pert  of  an  ongoing  effort  to  understand  the  processes 
people  employ  In  reading  technical  Material  and  the  ways  In  which  Information 
engineering  can  facilitate  those  processes.  This  study  provided  a  detained 
and  hierarchically  organized  Information  structure  as  part  of  a  / 

computer-assisted  Job-aiding  procedure  for  an  assembly  task.  The  central 
questions  were  "Where  people  have  easy  access  to  many  kinds  of  Information, 
what  Information  do  they  select  to  help  that  do  the  Job?  How  do  people  vary? 
How  does  Informatbn  selection  differ  across  sites  In  the  text  and  graphics?" 
The  results  of  this  experiment  provide  preliminary  answers  to  these  questions 
and  serve  as  the  basis  for  our  continuing  research  In  this  area. 


this  technical  report  presents  preliminary  results  of  on-going  research. 
A  second  report  containing  a  substantially  more  detailed  analysis  and 
discussion  of  results  is  currently  under  preparation. 


XnfoTMtlon  gaglaeerlng:  On-Uae  iail7>ii  \ 

of  Information  Search  and  Utilisation 


■wn  people  engaged  la  a  procedural  caak  have  ready  aeeoaa  to  a  wide  variety 
of  information,  what  information  do  they  seek  to  aid  thea  la  doing  the  job?  Bow 
do  people  differ  In  their  information  aearcb  end  how  doea  the  Information  affect 
their  porforaaaee?  Thl*  report  addreaaea  theae  queatloaa  through  doacrlptloa  of 
lafonatlon  aoaking  reapoaaea  node  during  an  assaably  teak,  for  which  aot  only 
haaic  inatructiona  (avrfaca  text)  bat  a  rich,  hierarchically  organised  aupplenental 
lafonatlon  structure  (Hypertext),  subject  to  aoar  control,  wan  laatantly  accessi¬ 
ble  by  naans  of  interactive  computer  terminal,  leap  oases  to  blocks  In  lafonatlon 
are  dlacnoaed  la  tana  of  problen  solving  atnteglea. 

lafonatlon  tnfiaaerlna  as  an  Anoroacb  to  Job  Udine 

The  approach  to  job  aiding  followed  here  night  beat  be  described  as  lnfor- 
nation  eng t nearing.  In  which  development  and  delivery  of  a  complex  lafonatlon 
structure  an  fully  Integrated  components  la  a  total  system  (Figure  1).  In  job- 
related  literacy,  such  as  reading  the  directions  for  a  procedural  task,  some  users 
any  aot  grasp  essential  technical  tans  or  spatial  orientations.  Tat  to  Include 
all  potentially  necessary  lafonatlon  la  tbs  text  would  Increase  considerably  the 
time  required  and  possibly  obscure  the  struetun  of  the  text  swan  for  these  who  do 
aot  need  that  lafonatlon.  One  approach  to  this  problem  is  to  provide,  via  conputer 
screen,  a  concise,  step-by-step  surface  text  of  directions,  pot  to  provide 
access  to  Hypertext,  a  detailed,  well-organised  information  structure  that 


display 
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may  include  a  wide  range  of  graphic  overviews  and  definitions  of  terns*  as  well 
as  an  easy  naans  of  reviewing  earlier  text  segnenta.  A  user  can  thus  call  up 
with  the  touch  of  a  light  pen  any  lnfomatlon  he/she  requires  but  need  not  vade 
through  lnfomatlon  he/she  doesn't  need. 

One  critical  feature  in  the  information  engineering  required  to  allow  these 
options  is  design  and  inplenentatlon  of  a  detailed,  well  organised  information 
structure.  The  second  critical  feature  is  design  and  implementation  of  a  delivery 
system  that  can  provide  instant  access  to  this  supplemental  information,  yet  return 
the  user  to  the  site  from  which  the  information  search  was  initiated.  These,  of 
course,  must  be  embedded  within  a  computer  system  capable  of  immediate,  high 
quality,  networked  responses. 

But  when  varied,  relevant  information  is  available,  what  do  people  select  and 
how  do  they  use  it?  Answers  to  this  question  can  contribute  to  our  knowledge  of 
how  best  to  design  job  aiding  procedures  and  to  our  understanding  of  how  people 
solve  (and  ultimately  can  be  helped  to  solve)  problems  of  comprehending  technical 
material. 

lesponses  to  Blocks  in  Information 

Even  when  instructions  in  a  task  are  informative  and  well  organized,  an 
information  block  (such  as  failure  to  understand  the  referent  for  e  technical  term 
or  the  Intended  orientation  or  relationship  of  two  parts)  nay  occur.  What  responses 
are  possible?  Ordinarily,  in  encountering  an  information  block  in  written  text 
a  reader  has  only  e  few  options — to  plunge  forward,  either  ignoring  the  block  or 
hoping  to  resolve  it  with  information  later  in  the  text,  or  to  move  backward  to 
review  earlier  information  that  might  provide  clarification. 

With  the  use  of  Hypertext,  however,  the  reader's  potential  responses  are 
Increased.  Figure  2  represents  these  options  schematically.  Faced  with  a  block 
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In  in  format  ion,  the  user  can  nova  forward  (in  which  caaa  wa  nay  not  know  a 
block  haa  occurred  except  through  perforaance  deficits),  move  backward  to  recheck 
earlier  text  or  Hypertext,  apve  down  to  check  the  definition  of  a  specific  technical 
tera  or  even  the  aeaning  of  a  familiar  word  In  this  context,  or  aove  up  to  a 
graphic  overview  of  the  aaseably  completed  to  this  step  or  a  graphic  overview  of 
the  completed  object. 

The  levels  of  the  hierarchy  thus  include  not  only  the  surface  text  of  directions 
but  a  more  detailed  level  of  definitions  of  terms  and  a  more  global  level,  in  this 
case  graphics  (though  In  other  content  the  global  level  might  be  verbal) .  This 
conception  of  the  hierarchical  structure  of  information  is  implicitly  rooted  In 
Frederlksen's  (1975)  analysis  of  discourse  processes,  though  those  notions  are 
operationalized  in  a  new  way. 

Information  Search  in  Technical  Literacy  as  Problem  Solving 

The  overall  goal  in  this  assembly  task  is  to  complete  the  correct  assembly  of 
the  miniature  loading  cart.  Subordinate  to  that  goal  are  the  subgoals  of  completing 
17  sequentially  arranged  steps,  each  guiding  transformation  of  the  object  from  one 
state  to  the  next.  (At  each  step  there  are  also  1  to  3  substeps  to  be  checked  off 
by  touching  the  light  pen  to  the  screen  as  completed,  but  our  analysis  focuses  on 
the  17  steps  of  the  surface  text.)  Subordinate  to  completion  of  a  given  step  are 
the  information-seeking  moves  undertaken  within  that  step.  These  can  Include 
forward  moves,  directly  completing  and  checking  off  each  step  and  subatep;  backward 
moves,  rechecking  Information  received  earlier;  upward  moves  (this  term  Is  an 
arbitrary  convention  but  psychologically  useful  In  visualizing  moves  within  the 
Information  structure  from  a  given  point  in  the  surface  text  to  graphic  information), 
end  downward  moves  to  secure  definition  of  terms  by  touching  the  light  pen  to  a 
word  in  the  surface  text  displayed  on  the  screen. 

While  the  overall  goal  and  the  subgoals  are  the  same  for  all  subjects  in  this 


1 


study,  ths  information  each  subject  requests  to  enable  himself /herself  to  complete 
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each  step  aay  vary  vldaly. 

The  focus  in  this  study  is  on  the  problem  solving  strategies  individuals  use 
in  overconing  inforaatlon  blocks.  Not  discussed  here  are  the  evaluation  and 
selection  processes  that  guide  a  given  user  in  selection  of  Inforaatlon  from  the 
inforaatlon  structure  provided,  from  his/her  own  information  structure,  and  from 
the  object  in  hand;  emphasis  is  on  the  directly  observable  moves  to  various  parts 
of  the  computer  information  system.  Newell  and  Slaon  (1972,  p.88)  described 
aspects  of  the  overall  organization  of  an  information  processing  system  in  this  way: 

1.  Input  translation,  producing  in  the  problea  solver  an  internal  representa¬ 
tion  of  the  problem  (in  this  case,  the  information  block)  to  be  solved.  "The  problem 
solving  then  proceeds  in  the  framework  of  the  internal  representation  thus  pro¬ 
duced— a  representation  that  may  render  problem  solutions  obvious,  obscure,  or 
perhaps  unattainable." 

2.  Selection  of  a  problem  solving  method;  "A  method  is  a  process  that  bears 
some  rational  relation  to  attaining  a  problem  solution,  as  formulated  and  seen  in 
terms  of  the  internal  representation."  In  this  case  the  available  methods  may 
Include  searching  the  total  Inforaatlon  structure  (prior  knowledge,  surface  text, 
Hypertext,  and  the  object  Itself),  drawing  inferences,  self -monitoring,  and  integra¬ 
ting  the  inforaatlon  bearing  upon  comprehension  of  a  particular  point. 

3.  Application  of  the  problea  solving  aathod  to  the  inforaatlon  block.  "At 
any  aoaent,  as  the  outcome  either  of  processes  Incorporated  in  the  method  itself 
or  of  more  general  processes  that  monitor  its  application,  the  execution  of  the 
asthod  aay  be  halted."  The  inforaatlon  block  asy  be  overcome,  apparently  overcoat, 
or  left  unresolved. 

4.  tear Quoins  if  necessary.  Newell  and  Slaon  note  (p.  88,  footnote)  that 
"The  continuous  flux  of  new  Inforaatlon  froa  the  environment  aay  offer  new  solution 
possibilities  or  demands  that  cause  the  problem  solver  to  Interrupt  (his/her)  cur¬ 
rant  activities  to  try  different  ones."  They  also  state  that  "when  a  asthod  is 
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terminated,  three  options  ere  open  to  the  problem  solver:  (e)  another  method  may 
be  attempted,  (b)  a  different  Internal  representation  may  be  selected  and  the 
problem  reformulated,  or  (c)  the  attempt  to  solve  the  problem  nay  be  abandoned", 
at  least  for  the  moment. 

5.  Generation  of  subgoals.  While  the  method  is  being  applied,  new  problems 
may  be  recognised.  "The  problem  solver  may  elect  to  attempt  one  of  these"  or 
may  set  aside  the  new  subgoals,  "continuing  instead  with  another  branch  of  the 
original  method." 

In  order  to  understand  the  problem  solving  processes  engaged  in  during 
computer-assisted  technical  literacy,  information  seeking  strategies  Initiated 
during  an  assembly  task  were  analysed  in  terms  of  these  categories:  forward  moves, 
rechecks  (backward  moves),  requests  for  graphics,  and  requests  for  dictionary. 
These  responses  were  examined  in  terms  of  overall  group  patterns,  individual 
patterns,  and  sites  in  the  text  that  tended  to  provoke  given  types  of  information 
search.  The  questions  to  be  answered  were: 

1.  What  is  the  relative  frequency  of  various  types  of  information 
requests  across  the  group? 

2.  How  do  individuals  vary  in  information  selection  and  use? 

3.  Do  various  sites  in  the  surface  text  provoke  different  numbers  and 
kinds  of  information  search? 

Method 


The  subjects  in  this  sample  were  not  a  homogeneous  or  random  sample  but  a 
wide  imaging  convenience  sample  of  13  adults.  They  ranged  in  background,  including 
eleetrleal  engineers,  stockboys,  secretaries,  graduate  students  and  teachers.  This 
diversity  curbed  any  tendency  to  general ire  too  freely  from  a  narrow  sample,  end 
presented  a  strong  test  of  the  ability  of  the  Hypertext  system  to  respond  to  the 
information  needs  of  users  ranging  widely  in  background. 


The  materials  used  Include  17  frames  of  directions  for  msooably  of  a  Miniature 
loading  cart  to  be  assembled  from  blocks.  This  surface  text,  presented  on 
a  computer  display  screen,  was  written  at  fifth  grade  level  and  presented  each 
step  clearly  and  concisely,  as  shown  in  Figure  3.  Bach  text  frame,  however,  gave 
the  viewer  the  option  of  rechecking  text  seen  earlier  or  branching  into  Hypertext. 

By  touching  with  a  light  pen  almost  any  substance  word  on  the  screen,  the  viewer 
called  up  a  frame  (cf.  Figure  4)  that  defined  the  term  or  explained  its  meaning 
in  that  context;  often  the  definitions  were  accompanied  by  small  illustrations  of 
the  part  or  operation  defined. 

The  viewer  also  had  the  option  of  viewing  several  types  of  color  graphics. 

By  touching  one  of  the  boxes  centered  below  the  directions  for  each  substep,  the 
user  could  view  an  illustration  of  the  loading  cart  aa  it  should  appear  at  that 
step  in  the  assembly  process  (Figure  5).  Or,  by  touching  a  box  in  the  upper  left 
corner,  the  user  could,  at  any  point  in  the  program,  branch  to  a  graphic  depicting 
the  completed  object.  Touching  certain  sections  of  that  graphic,  such  as  the 
axle  assembly,  shown  in  front  view,  could  call  up  a  more  detailed  "blow-up"  of  that 
component.  For  some  subassemblies  or  pares  it  was  also  possible  to  request  another 
view  (front,  back,  side,  up-side-down) . 

The  detached  display  screen  was  linked  to  Hazeltlne's  TXCCIT  (Time-shared 
Interactive  Computer  Controlled  Information  Television)  •  A  central  feature  of  the 
system  for  this  purpose  was  the  Hypertext  ™  Display  System.  Technically,  this  term 
includes  the  organisation  of  both  surface  text,  and  the  supplemental  graphics  and 
definitions,  though  we  will  at  times  speak  of  contrasts  between  text  and  Hypertext . 
Fuller  description  of  this  system  is  given  in  Stone  and  McMlnn  (1982)  and  Stone, 
Israelite,  Hedrick,  and  Hutson  (1983). 
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Data-gatherlng  Procedures 

Each  subject,  tasted  individually,  vaa  seated  at  a  computer  display  console 
equipped  with  a  light  pen.  The  parts  needed  to  assemble  the  miniature  loading 
cart  were  spread  out  on  the  table.  The  examiner  explained  the  purpose  of  the 
study  and  led  the  subject  through  the  introductory  phase  of  the  program,  which 
explained  and  Illustrated  the  information  search  options  available.  Then,  as 
the  subject  progressed  through  the  steps  of  the  program,  the  examiner  made  notes 
on  his/her  performance. 

An  Internal  tracking  program  recorded  and  later  printed  out  the  sequence  of 
numbers  of  the  frames  viewed  by  each  subject.  The  sequence  of  moves  for  each 
subject  was  transformed  into  a  schematic  profile  for  each  subject,  and  the  types 
of  information  moves  were  tabulated  by  subject  and  by  frame. 

Data  Analysis  Procedures 

Operationally  defined,  the  key  variables  were: 

Directness  -  100  minus  the  number  of  forrard  and  backward  steps: 

Use  of  Hypertext  -  Number  of  requests  for  graphic  information  or 

for  dictionary 

Rechecks  -  Starting  point  for  long  or  short  backward  moves  to  text 
or  Hypertext 

Based  on  the  conceptualisation  of  information  blocks  and  information  search 
strategies  discussed  earlier,  the  computer's  Internal  tracking  records  of  the 
sequence  of  frame  numbers  viewed  were  transformed  into  visually  salient  graphic 
representations  of  each  subjects'  performance.  This  facilitated  explicit  descrip¬ 
tion  of  each  move  by  each  subject  yet  made  it  easy  to  detect  the  larger  patterns 
within  which  individual  moves  were  embedded.  The  next  step  was  to  tabulate  relative 
frequency  of  levels  of  directness  and  various  information  requests  for  each 
subject  and  across  the  whole  group.  A  similar  procedure  was  used  to  examine 
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differences  between  prograa  frames  In  the  reletive  frequency  of  various  Informa¬ 
tion  moves.  Described  later  is  the  development  of  a  coding  system  for  cluster 
analysis  (to  be  conducted  after  more  subjects  are  observed)  in  order  to  develop 
a  taxonomy  of  frame  types. 

Results 

The  results  of  this  analysis  are  organized  in  terms  of  1)  overall  patterns 
of  information  search  and  utilization,  2)  individual  variability,  and  3) 
analysis  of  the  program  steps  that  provoked  various  information  requests. 

What  is  the  relative  frequency  of  various  information  requests,  across  the  group? 

People  vary  in  the  background  and  strategies  they  bring  to  a  task  like  this, 
and  they  va  v  in  the  information  they  seek  during  the  task.  The  major  dimensions 
along  which  they  differ  in  performance  are  directness,  use  of  Hypertext,  and 
rechecks.  It's  tempting  to  think  of  directness  (use  of  a  minimum 
number  of  forward  moves)  as  efficient,  but  it's  not  really  efficient  to  plunge 
ahead  blindly  if  needed  information  is  readily  available.  Use  of  Hypertext  and 
a  few  recursions  may  aid  performance.  Heavy  use  of  recursions,  on  the  other  hand 
may  indicate  problems. 

Across  the  whole  group  (Table  1)  there  were  238  information  requests,  for  a 
mean  of  17.3  information  request  moves  per  user.  The  most  frequently  requested 
category  was  graphics,  with  129  requasta  (541  of  the  total),  followed  by  dictionary 
requests,  73  (31Z).  The  remaining  15Z  of  the  requests  took  the  form  of  rechecks 
of  text  (24  requests)  or  Hypertext  (12  requests) . 

Across  the  group  the  average  figure  for  directness  (100  -  the  total  number 
of  forward  and  backward  moves)  was  72.4.  The  minimum  number  of  forward  moves 
for  the  17  fraams  was  expected  to  be  17  (though  one  subject  viewed  only  16  fraams). 


This  would  yield  a  directness  Index  of  83,  which  was  attained  by  several  subjects. 
On  the  average,  subjects  took  eleven  more  stoves  forward  and  backward  than  the 
minimum  needed  to  complete  the  task,  but  one  took  thirty  moves  more  than  the 
minimum,  and  several  took  only  the  minimum. 

How  did  Individuals  vary  In  Information  selection  and  use? 

One  variable  on  which  subjects  hardly  differed  was  success  on  the  task. 

The  directions  were  designed  to  be  clear  and  readable,  yet  previous  studies  using 
the  same  surface  text  (Stone  &  Clock,  1981)  showed  that  two-thirds  of  the  subjects 
completed  the  task  with  some  uncorrected  errors.  In  this  study,  with  the  addition 
of  Hypertext,  almost  no  errors  were  uncorrected  by  the  end  of  the  task.  Subjects 
varied,  however.  In  the  strategies  they  used  In  selecting  Information  they  needed 
to  succeed  on  the  task. 

Schematic  representations  of  performances  by  several  subjects  are 
displayed  In  Figure  6.  The  center  lines  represent  the  17  steps  of  the  surface 
text,  the  concise  sequential  set  of  Instructions  for  assembling  the  loading 
cart.  Requests  for  graphics  are  represented  as  moves  up  to  the  boxes  above 
the  line,  and  requests  for  definitions  of  terms  are  represented  as  moves  down 
to  circles  below  the  line.  Rechecks  are  represented  as  moves  backward  (right 
to  left). 

The  subjects  shown  In  Figure  6  are  relatively  direct  in  approach.  Subject 
109  moves  straight  through  the  taskonthe  level  of  the  surface  text,  but  at 
the  end  checks  back  at  two  points,  apparently  to  view  the  graphic  displays  there. 
Subjects  102,  119,  110,  115,  and  111  make  few  or  no  rechecks  but  make  increasing 
use  of  Hypertext.  While  subject  111  has  27  requests  for  Hypertext  (17  for  graphics, 
10  for  dictionary),  as  contrastad  with  2  by  subject  109,  his/her  forward  progress 
is  equally  direct. 


Pattern*  for  the  subjects  shown  in  Figure  7  show  low-moderate  use  of 
rechecks  and  varying  us*  of  Hypertext.  Even  in  soae  of  these  subjects'  pro¬ 
files  there  is  a  suggestion  of  increased  action  around  fraaes  5,  11,  and  16. 

Subject  117  requests  definition  of  terns  at  frame  5,  moves  on  to  frame  6,  but 
rechecks  several  points  in  text  or  Hypertext,  apparently  resolves  the  problem 
and  moves  on.  Subject  108  does  not  stop  at  that  point,  but  on  reaching  the 
last  frame  makes  a  long  sweep  back  to  frame  5,  then  moves  forward  to  frame  11, 
checks  one  graphic  and  on*  definition,  then  checks  out,  having  successfully 
completed  the  task. 

Nor*  complex  and  less  direct  patterns  are  seen  for  subjects  103,  116, 
and  105  (Figure  8).  For  103  and  116  the  pattens  differ  more  in  degree  than  in 
kind  from  the  group  just  discussed;  some  problems  around  fraaes  4-5,  soae  near 
frames  10-11,  same  rechecks  back  to  the  earlier  trouble  spot,  and  a  long  sweep 
back  from  the  last  frame.  Subject  105  is  perhaps  the  most  complex,  with  recur¬ 
sions  within  recursions.  Except  for  a  swirl  of  text  and  Hypertext  rechecks 
around  frames  4-5 ,  apparently  resolving  the  problem,  he/she  proceeds  with  only  a 
few  asides  to  the  next  to  the  last  frame,  checks  a  term,  moves  back  a  few  frame* 
to  check  some  graphics,  moves  forward  one  fraam,  checks  a  graphic,  then  moves  on 
for  on*  more  frame.  At  that  point,  however,  he/she  reverses  direction,  retuning 
to  check  the  text  on  frame  10,  moves  forward  two  frames,  then  reverses  to  recheck 
a  term  on  frame  11,  then  checks  out,  job  completed. 

He  present  here  simply  a  description  rather  than  an  evaluation  of  the  effective¬ 
ness  of  users*  search  strategies,  but  even  these  preliminary  findings  suggest 
several  tactics  for  aiding  performance  or  leaning.  Where  there  are  clearly  defined 
effective  and  ineffective  search  strategies,  an  on-line  Diagnosis  subroutine  can 
provide  visually  salient  feedback  such  as  tha  users'  profile  or  napping  of  that 
individual's  performance  against  a  background  of  typical  or  ideal  performance. 
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lt  can  also  ba  programed  to  Intervane  in  one  of  several  ways,  triggered  by  the 
pattern  of  infomation  moves  in  a  given  area  of  the  text  and  some  if-when-then 
statements.  Such  interventions  can  range  from  gentle  advisor  messages:  "Are  you 
lost?  If  you'd  like  to  view  the  completed  object,  touch  the  green  box")  to  more 
direct  measures  (locking  in  or  out  certain  portions  of  the  information  structure 
for  a  given  user,  either  to  Induce  new  strategies  or  to  prevent  disasters  in 
safety  or  security) . 

In  this  study  all  subjects  succeeded,  but  they  succeeded  in  different  ways, 
making  use  of  the  Information  they  felt  they  needed  at  various  points.  The  total 
number  of  information  requests  and  the  kinds  of  information  requested  varied  across 
subjects. 

Which  program  steps  provoke  search? 

Some  steps  in  the  set  of  directions  for  this  assembly  task  appeared  to  aet 
off  more  searches  and  different  kinds  of  seraches  than  others.  As  shown  in 
Table  2  and  Figure  9,  the  total  number  of  searches  springing  off  a  given  program 
step  ranged  from  2  for  step  15  to  36  for  step  5.  In  general,  steps  that  provoked 
a  number  of  dictionary  requests  or  text  rechecks  were  the  same  as  those  that 
provoked  graphic  requests,  signalling  a  general  need  for  clarification  that  some 
users  satisfied  In  one  way  and  some  in  another.  Step  5  provoked  both  17  requests 
for  graphics  (several  of  them  repeats  by  the  same  user),  most  for  graphic  10, 
the  component  as  completed  to  that  step,  and  14  for  dictionary.  Step  11  elicited 
16  requests  for  graphics  and  16  for  dictionary.  Graphic  27,  an  overview  of  the  com¬ 
pleted  object,  was  requested  most  often  at  frames  4  and  16.  As  a  rule ,  there  were 
more  requests  for  graphics  than  for  dictionary  on  a  given  frame,  but  on  step  16 
a  substantial  number  of  requests  were  made  for  clarification  of  the  term  "grooves." 

Some  steps  (notably  5  and  10)  provoked  more  text  rechecks  than  most,  as  though 


the  nav  step  atdt  users  rsaliss  thsc  thsrs  vss  something  earlier  they  hadn't 
understood.  For  step  5,  however,  e  masher  of  the  rechecks  to  step  4  and  to  the 

related  graphic  were  repeats  by  the  ease  uaar.  For  frame  17  the  rechecks  were 

In  soae  cases  responses  to  the  invitation  on  that  fraae:  "This  completes  the 

asseably  of  the  loading  cart.  You  aay  go  back  over  any  part  of  these  instructions 

to  check  your  work,  however."  Both  the  snail  number  of  text  rechecks  and  the 

uneven  proportion  that  were  repeats  by  the  same  user  suggest  that  while  this  is  a 

dimension  useful  In  describing  Individual  differences,  use  of  text  rechecks  on  a 

given  step  as  a  signal  of  program  problems  would  require  a  larger  sample. 

Across  the  group  as  a  whole,  the  largest  numbers  of  information  searches 
emanated  from  frames  5  and  11,  with  frsmea  16  (mostly  definitions)  and  10 
(mostly  graphics)  not  far  behind.  There  were  very  few  information  requests 
initiated  at  frames  2,  6,  IS,  and  7. 

It  is  also  possible  to  analyze  more  carefully  the  information  requests 
at  any  point.  Fraae  5,  for  example,  directed  the  user  to  form  the  sxle 
asseably.  At  this  frame  the  graphic  selected  at  least  once  by  10  of  the  13 
subjects  was  the  illustration  of  that  step  when  completed.  One  person  rechecked 
the  graphic  of  the  prior  step  and  two  requested  e  view  of  the  finished 
object.  The  terns  for  which  clarification  was  requested  by  one  person  were 
"attached,"  "end  groove  of  column  on,"  and  "end  groove  of  column  two."  Terms 
requested  by  three  people  were  "angle  block”  and  "are  oriented  correctly." 

An  on-line  Debug  subroutine  of  the  computer  progrsa  can  provide  grephlc  and 
quantitative  data  as  input  in  the  human  judgement  of  the  best  fix-up  strategy  for 
observed  trouble  spots.  If  the  nvaber  of  information  requests  for  a  given  program 
step  is  small,  it's  usually  best  to  lsave  it  alone.  Only  a  few  people  Meded  the 
help  available  there,  but  for  those  who  needed  it  the  help  aay  have  been  critical. 
The  cost  of  developing  and  storing  tha  supplemental  information  in  Hypertext  is 
likely  to  be  far  nailer  than  the  cost  of  mistakes  made  in  technical  operations. 
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If  the  number  of  requests  for  Information  at  a  given  program  stap  ia 
large,  it'a  useful  first  to  determine  whether  the  requests  taka  the  form  of 
graphics  and/or  dictionary  or  of  text  rechecks.  If  the  former,  the  software 
designer  msy  decide  to  import  the  information  into  the  surface  text  (adding 
a  graphic  detail  or  defining  a  term  In  context)  or  may  decide  to  leave  it  as  is, 
reasoning  that  if  no  more  than  a  quarter  of  the  users  need  that  information,  it's 
still  easily  available  to  them,  while  leaving  the  surface  text  uncluttered  for 
those  who  don't  need  that  Information.  Number  of  errors  or  rechecks  following 
this  step  may  also  be  considered. 

A  large  number  of  rechecks  in  a  given  area  of  the  program  usually  signals  a 
problem,  but  the  action  to  be  taken  depends  upon  the  nature  of  the  problem.  If 
the  total  is  a  reflection  of  repetitions  on  the  same  frames  by  only  a  few  people, 
it's  probably  an  individual  problem  of  lack  of  knowledge  or  lack  of  assurance. 

The  problem  may  be  alleviated  indirectly  if  requests  for  Hypertext  on  the  same 
frames  lead  to  the  changes  described  earlier.  If  rechecka  in  a  given  area  are 
made  by  more  than  a  few  people,  it  is  useful  to  examine  not  only  the  frames  that 
are  the  starting  point  and  ending  point  for  the  recheck  but  the  relationship  between 
them.  Is  there  anything  misstated  or  misinterpreted  in  the  first  frame?  Does  the 
problem  become  apparent  only  in  the  next  frame?  Or  is  there  a  new  term  for  the  samt 
part  mentioned  earlier?  As  stated  earlier,  text  rechecks  may  be  more  useful  in 
flagging  possible  problems  than  in  spelling  out  the  solutions. 

Although  this  informal  analysis  was  based  on  only  a  few  users,  it  indicates 
that  some  sites  in  the  surface  text  provoke  more  information  requests  than  others. 

It  also  points  up  the  feasibility  of  a  strategy  for  responding  to  a  serious 
practical  problem:  the  need  for  early  dabugglng  of  software.  As  Bunyard  and 
Coward  (1982)  point  out,  errors  (or  less  than  optimal  solutions)  detected  late  in 
the  development  of  software  cost  far  more  to  correct  than  those  detected  early. 
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The  complex  Information  structure  available  to  users  in  this  study  appears 
not  only  to  fadlitste  performance  through  Individual  control  of  information 
needs,  but  to  fecilltete  debugging  by  pointing  out  both  the  sitea  where  more 
information  is  required  end  the  kinds  of  informstion  sought.  These  patterns 
can  readily  be  diagnosed  on-line  and  accumulated  over  a  modest  number  of  subjects, 
flagging  trouble-spots  and  suggesting  the  needed  fix-up  strategies.  Some  changes 
might  be  made  even  while  the  software  is  still  in  the  development  phase,  substantially 
reducing  life-cycle  costs  (Grove,  1982)  and  minimising  performance  errors.  Such 
a  Debug  subroutine  could  form  the  core  of  a  software  quality  assurance  program,  as 
described  by  Baker  &  Fisher  (1982) . 

Development  of  a  Taxonomy  of  Information  Search  Patterns 

This  segment  discusses  the  preparation  for  an  aspect  of  analysis  that  will 
not  be  conducted  until  profiles  of  more  subjects  have  been  analyzed.  The  Intent 
at  that  time  is  to  conduct  cluster  analysis  in  order  to  determine  a  taxonomy  of 
types  of  frames,  described  in  terms  of  the  information  moves  they  provoke. 

The  first  step  in  that  process  is  to  develop  a  coding  system  to  describe  in 
a  limited  number  of  dimensions  the  complexity  of  the  total  activity  (not  just 
a  single  move,  but  all  the  activity)  undertaken  by  a  given  subject  on  a  given  frame. 

Each  subject's  activity  on  a  frame  is  described  in  terms  of  three  digits, 
each  representing  one  dimension  of  the  taxonomy.  Presented  in  Table  3  are  the 
three  dimensions:  Numbers  of  passes  (forward  and  backward)  a  subject  makes 
through  that  frame  either  during  Initial  progress  or  later  for  rechecks;  use 
of  Hypertext;  and  beginning  or  end  of  recheck. 

To  illustrate  the  application  of  the  coding  system,  consider  frames  1 
through  6  for  subject  105.  The  first  frsme  is  coded  as  100  (1  pass,  no  Hypertext, 
no  rechecks),  the  second  as  103  (1  pass,  no  Hypertext,  the  end  of  a  1  step  recheck), 
the  third  as  345  (3  passes,  1  request  for  dictionary,  the  beginning  and  rad  of 
a  1-step  recheck).  Frsme  4  is  coded  as  315  (3  passes,  1  request  for  graphics. 
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beginning  and  and  of  2  or  nor*  rechecks  (long  or  ahort).  .  Fraa  5  is  eodad  as  787 
(7  passes,  both  kinds  of  graphics,  nora  than  1  of  either,  beginning  of  more  than 
1  recheck).  And  Frame  6  is  simply  100  (1  pass,  no  Hypertext,  no  rechecks) ; 
coders  learn  to  treasure  such  sinple  items.  By  coding  and  analysing  frames  in 
this  way  it  should  be  possible  to  locate  the  clusters  of  similar  responses  to  frames. 

Final  Comments 

This  report  is  part  of  an  ongoing  effort  to  understand  the  processes  people 
esiploy  in  reading  technical  material  and  the  ways  in  which  information  engineer¬ 
ing  can  facilitate  those  processes.  This  study  provided  a  detailed  and  hierarchi¬ 
cally  organized  information  structure  as  part  of  a  computer-assisted  job  aiding 
procedure  for  an  assembly  task.  The  central  questions  were  "When  people  have 
easy  access  to  many  kinds  of  information,  what  information  do  they  select  to 
help  them  do  the  job?  How  do  people  vary?  How  does  Information  selection  differ 
across  sites  in  Che  text?" 

Subjects  as  a  group  selected  graphic  information  most  often,  dictionary  next, 
and  sometimes  made  rechecks  of  text  or  nontext  information.  Individuals  varied, 
however,  in  the  directness  of  their  approach,  the  frequency  of  their  requests  for 
graphics  or  dictionary,  and  their  use  of  long  and  short  rechecks.  Yet  at  a  higher 
level  of  generality  some  common  patterns  were  observed.  Sites  in  the  text  also 
differed  widely;  some  were  the  starting  point  for  far  acre  rechecks  and  requests 
for  Hypertext  than  were  others. 

Performance  was  noticeably  better  than  in  previous  studies  using  the  same 
materials  but  without  the  Hypertext  feature  of  ready  access  to  e  hierarchically 
organised  information  structure.  This  suggests  that  when  people  can  get  the 
lnformetlon  they  need  when  they  need  it,  performance  on  procedural  tasks  is 
facilitated.  But  people  vary  in  the  information  they  feel  they  need,  so  it  may 
be  advisable  to  have  more  information  (and  more  forms  of  information)  available 
than  any  one  person  is  likely  to  need,  yet  to  keep  the  surface  text  uncluttered 
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In  order  to  avoid  Information  overload  for  thoae  who  don't  need  that  Information. 

There  ere  several  levels  of  generality  at  which  the  procedures  described  here 
may  be  useful  (apart  from  their  central  function  in  job  aiding):  early  debugging 
of  software,  systematic  objective  study  of  the  design  of  job  elding  procedures, 
elding  the  study  of  processes  involved  in  comprehending  technical  material,  and 
aiding  the  study  of  problem  solving  in  a  specific  technical  context.  For  example, 
the  objective  on-line  date-gathering  can  be  complemented  by  thlnk-aloud  or 
stimulated  recall  techniques  to  understand  how  individuals  conceptualise  a  par¬ 
ticular  problem  and  organise  relevant  information. 

In  terms  of  the  aspects  of  information  identified  by  Newell  and  Simon  (1972) , 
we  have  little  direct  information  about  input  translation,  an  aspect  that  will 
deserve  a  closer  look  after  the  more  easily  observed  aspects  are  better  mapped. 

We  have  indirect  evidence  about  selection  of  problem  solving  methods;  analysis  by 
frames  show  that  the  types  of  information  requests  provoked  differed  across  frames. 
We  have  a  clear  record  of  those  aspects  of  application  of  problem  solving  method 
msnlfest  in  requests  for  various  kinds  of  information  (though  no  documentation  of 
the  more  internal  aspects  such  as  inference  and  Integration  and  linkage  to  prior 
knowledge).  The  same  data  base  indicates  that  subjects  often  regroup  (request 
more  information)  when  the  problem  of  comprehending  a  particular  frame  resists 
initial  attempts  at  solution.  On  the  point  of  subgoals,  our  evidence  is  only  sug¬ 
gestive.  Bursts  of  requests  for  rechecks  or  Hypertext  may  suggest  that  the 
individual  sets  as  subgoals  clarification  of  one  or  more  points  before  he/ she  can 
usefully  attend  to  the  larger  goal  of  successfully  completing  that  step.  Protocols 
that  show  an  individual  moving  through  a  trouble  spot  but  later  making  a  long 
recheck  to  that  spot  may  illustrate  Newell  and  Simon's  point  that  problem  solvers 
sometimes  set  aside  new  subgoals  (in  this  case,  a  desire  for  greater  clarity  about 
sons  terms,  spatial  orientations  or  relationships  between  subcomponents)  but  may 


later  have  to  return  to  resolve  that  problea.  Perhaps  aost  clearly  suggestive 
of  the  generation  of  subgoals  Is  the  protocol  of  subject  105,  in  which  the  indivi¬ 
dual  on  more  than  one  occasion  reversed  direction,  or  116,  who,  in  the  middle  of 
a  recheck,  sent  out  requests  for  Hypertext. 

Experts  on  problea  solving  such  as  Newell  and  Slaon  (1972)  or  Shulaan  and 
Elstein  (1975)  Invested  many  years,  close  analysis,  and  auch  thoughtful  analysis 
and,  undoubtedly,  reconceptualization  before  presenting  foraal  models  of  the  process. 
It  would  take  acre  chutzpah  than  the  authors  of  this  report  collectively  possess 
to  present  a  foraal  model  at  this  point.  The  results  we  present  here  are  only  a 
beginning,  and  we  have  elected  to  start  froa  the  outside  with  objectively  observ¬ 
able  behavior,  then  gradually  to  circle  closer. 

Some  major  patterns,  however,  seem  to  be  emerging.  It  seeas  legitimate,  to 
describe  and  interpret  these  behaviors  as  problem  solving  in  a  specific  technical 
context.  That  context  to  some  degree  defines  the  problea  space  add  thus  the 
solutions  likely  to  be  tried,  yet  individuals,  differing  in  background  and  having 
access  to  many  kinds  of  relevant  information,  varied  in  the  strategies  they  applied. 
Nonetheless,  there  were  some  patterns  across  the  group  (though  none  that  were 
universal).  The  more  obvious  patterns  were  linked  to  difficulties  present  in  certain 
segments  of  the  text,  as  indicated  by  the  greater  nuaber  of  information  searches 
around  certain  frames  or  the  greater  tendency  to  seek  definitions  for  some  kinds 
of  information  blocks,  graphics  for  others. 

We  hope  that  the  procedures  and  analyses  described  here  will  prove  useful  both 
in  enhancing  job  performance  and  training  and  in  exploring  the  Information  search 
and  utilization  processes  used  in  solving  important  practical  problems. 


Individual  Variability  In  Infbrmtlon  Search  Patterns 


If  long  text  rbchecks  Mere  Interrupted  to  seek  Hypertext  Information,  then  resumed  leftward,  a  second 
recheck  was  tallied;  Includes  long  and  short  rechecks  for  text  or  Hypertext. 


Tabic  2 


Frequency  of  Information  Requests  Initiated 
at  Each  Program  Step' 


Frame 

Graphic 

Requests 

Dictionary 

Requests 

Start  of  Text 
Recheck 

Start  of  Non¬ 
text  Rechecks 

Total 

1 

9 

2 

0 

0 

11 

2 

3 

0 

1 

0 

4 

3 

4 

3 

1 

0 

8 

4 

12 

3 

3 

0 

18 

5 

17 

14 

4 

1 

36 

6 

3 

1 

1 

1 

6 

7 

8 

1 

2 

0 

11 

8 

4 

1 

0 

3 

8 

9 

12 

3 

0 

0 

15 

10 

13 

5 

3 

1 

22 

11 

16 

16 

0 

0 

32 

12 

8 

2 

1 

1 

12 

13 

8 

0 

2 

0 

10 

14 

9 

4 

0 

1 

14 

15 

0 

1 

0 

1 

2 

16 

3 

17 

1 

1 

22 

17 

0 

0 

5 

2 

7 

129 

73 

24 

12 

238 

1  Analysis  based  on  the  first  13  subjects 


Tabic  3 


Taxonomy  of  Soar  eh  Activity  by  Subjects  at  Each  Frees 


fit at  dialt 

Mo.  of  Paaaaa  Forward  or  Backward 

0  •  nous 

1  •  one  tins 

2  •  two  tlaas 

3  -  throe  tlaas 

4  -  four  tlaas 
3  -  five  times 

6  ■  six  tlaas 

7  -  seven  tines 


Second  dialt 
Use  of  Hypertext 

0  ■  none 

1  -  graphics  (1),  including  repeats 

2  -  graphics  (2) 

3  -  graphics  (3  or  aore) 

4  -  dictionary  (1) 

5  ■  dictionary  (2) 

6  ■  dictionary  (3  or  aora) 

7  -  one  of  each 

8  -  both,  aore  than  1  of  either 


Third  dixit 

Beginning  or  End  of  Text  or  Hypertext  Recheck 

0  ■  not  beginning  or  end  of  recheck 

1  ■  beginning  of  rechack  of  1  step 

2  •  beginning  of  recheck  of  2  or  aore  steps 

3  ■  end  of  recheck  of  1  step 

4  *  end  of  recheck  of  2  or  aore  steps 

3  •  beginning  and  end  of  2  or  aore  re checks  of  1  step 

6  ■  beginning  and  end  of  2  or  aore  rechecks,  at  least  1  of  which 

is  2  or  aore  steps  long 

7  -  beginning  of  2  or  aore  rechecks,  short  or  long 

8  ■  end  of  2  or  aore  rechecks,  short  or  long 
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Figure  9.  Relative  frequency  of  various  Information  requests  at  each  step 
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